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The data rate of a multilevel digital FM system is optimized subject 
to fixed RF bandwidth, signal-to-noise ratio, and output error rate. The 
possibility of optimizing such a system was first considered by J. R. 
Pierce at Bell Telephone Laboratories. He made the observation that it 
is possible to send many levels slowly or fewer levels rapidly for an FM 
wave of fixed RF bandwidth and error rate, and that there must be a choice 
of signaling rate and number of levels that optimize the data rate. The 
rigorous treatment of this problem is the subject of this paper. The mathe- 
matical model we analyze uses frequency-shift keying at the transmitter 
and ideal discrimination detection with an integrate-and-dump circuit as 
the post-detection filter. Our results are exhibited graphically showing the 
various dependencies among the pertinent system parameters. 

I. INTRODUCTION 

In this paper we optimize the information rate (subject to certain 
constraints) of a multilevel digital FM system. This problem of 
delivering the maximum information through an FM system has 
recently been formulated by J. R. Pierce. 1 Specifically, he considered 
how one should choose the baseband signaling rate and the number 
of levels to get the most information through the channel, subject to 
fixed bandwidth, fixed RF signal-to-noise ratio, and fixed output error 
rate. This optimization has recently been carried out under the assump- 
tion that the conventional FM receiver can be linearized. 2 Small-noise 
linear FM theory is satisfactory when analyzing analog systems, but 
has its well known pitfalls in digital applications. 

The purpose of this paper is to reexamine this problem more rig- 
orously, paying particular attention to the anomalies (clicks) which 
can result from the nonlinear character of the receiver. In order to 
do this we must choose a particular mathematical model for digital 
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FM which is amenable to analysis. Such a model uses frequency-shift 
keying (FSK) at the transmitter and ideal discrimination detection 
with an integrate-and-dump circuit as the postdetection filter. The 
noise at RF is assumed to possess gaussian statistics. Although realizable 
FM systems do not exactly conform to this ideal mathematical model, 
we feel that the results predicted with the use of this model are appli- 
cable to real FM systems. In any case, the numerical results agree 
well with those derived from the linear theory. According to our present 
calculations, this is due to the circumstance that the optimum number 
of levels leads to small enough deviations so that the contribution 
of the clicks to the error rate can be neglected. 

II. ANALYSIS 

Consider an n-level FSK communication system with a sample rate 
JV = 1/T, square-wave modulation, and a level separation (in frequency) 
A/. Such a system would yield a data rate R given by 

R = N log 2 n = 1.443 N In n bits/s, (1) 

and, according to Carson's rule, occupy a bandwidth* 

B = N + (n - 1)A/. (2) 

The FM signal plus gaussian noise enters a receiver consisting of an 
ideal RF filter (bandwidth B), limiter, discriminator, integrator (in- 
tegration time T), and sampler (sampling rate N). The sampler out- 
puts are simply the successive values of the instantaneous phase of 
the modulated wave following each (rectangular) modulation pulse, 
and would be separated by multiples of 

A<6 = 2ir -77 radians (3) 

N 

in the absence of noise. 

The simplicity of the present system (that is, the finite-time integrator 
post-detection filter) has permitted a fairly rigorous determination of 
the probability of error for high RF signal-to-noise ratio. 4 It is shown 
in Ref. 4 that the parameter A<£ given in equation (3) plays a very 
important role in the theory of error rates for digital FM. In particular, 
it is known that if Ac/> < t (or equivalently, Aj/N < \), then it is 
the smooth noise at the baseband output which determines the error 



* Comparison with the exact FSK spectra for n = 2, 4, 8 suggests that this 
approximation is valid for present purposes. 3 
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rate; while if A<p > tt {Aj/N > %), then the clicks dominate, which 
is the basic reason for the probability of error taking on different forms 
in these two cases. 

The optimum systems considered here are shown to correspond to 
the Aj/N < % case, for which clicks are unimportant. Therefore we 
take the probability of error* P as given by twice equation (17a) of 
Ref. 4, with <f> -+ A<f>/2 = x A//A7; 



" ,tl if 



(2**jV / A/ 



COS I W ~Tj 



exp[-2psin 2 (|^j 



P »l, f <|, (4) 

and subsequently verify that Aj/N is indeed less than \ for the re- 
sulting optimum systems. Here p is the RF signal-to-noise ratio in 
the frequency band B. We treat the asymptotic approximation (for 
large p) of equation (4) as an equality in the following. 

For fixed error rate P and RF signal-to-noise ratio p, equation (4) 
determines Aj/N. Rewriting equation (2), 

!-! + <»- i)f; (5) 



substituting equation (5) into equation (1), 
R 1.443 Inn 



B "i + (.-i)f 



bits/ cycle. (6) 



We set the derivative of equation (6) equal to zero, determining the 
optimum number of levels n and maximum rate R ■ 

n„(lnn - 1) = jtj^ - 1. (7) 

fin _ 1-443 /gs 

B " n n {Aj/N) 
Alternatively, once the optimum number of levels n has been de- 



* For multilevel output samples, most errors will be to adjacent levels. Assuming 
that something like the Gray code is used, the symbol probability of error P of 
equation (4) will be approximately the bit probability of error for the final recon- 
structed binary signal. 
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termined via equations (4) and (7), we may express the other parameters 
of the (optimum) system in terms of n only: 

1 



^ = 

N n (ln n - 1) + 1 ' 



B 
B 



= 1 



(9) 

.443 lnn + — - 1 bits/ cycle, (10) 

n n In n n 



(11) 



N n (lnn — 1) + 1 
Note that the restriction Af/N < \ implies via equation (7) that 

n ^ 4. (12) 

Finally, the Shannon capacity for the RF channel is 



„ = 1.443 In (1 + p) bits/ cycle. 
B 



(13) 



III. RESULTS 

Figures 1 to 7 illustrate the parameters of optimum multilevel 
FM systems using a finite-time integrator as a post-detection filter 
for two representative error rates (P = 10~ 6 , 10" 8 ). 

The solid curves of Fig. 1 show the optimum number of levels n 
versus the RF signal-to-noise ratio in dB, 10 log™ p, for the two values 
of P. The curves terminate at n = 4, according to equation (12). 









p= 


6 /f 




























^p,= 


I0" a 




~*0^ 









10 20 30 40 50 

10 LOG| p-RF SIGNAL-TO-NOISE 

RATIO IN DECIBELS 



Fig. 1 — Number of levels for maximum data rate versus RF signal-to-noise ratio. 
Dashed lines indicate Bmall-angle approximations. 
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Fig. 2 — Bandwidth expansion factor for maximum data rate. 

n increases rapidly as p increases, for fixed P. The small-angle approx- 
imation for the trigonometric functions in equation (4) is shown by 
the dashed curves of Fig. 1; in this approximation changing P simply 
translates the curves of Fig. 1 horizontally. This is a reasonable ap- 
proximation for the smallest n permitted [by equation (12)], for the 
values of P of interest here. 

Figures 2, 3, 4, and 5 show optimum system parameters plotted 
against two horizontal scales: 

(i) 10 log 10 p — the RF signal-to-noise ratio in dB. Two plots are 
shown, for P = 10 _a , 10~ 8 . Using the small-angle approximation in 
equation (4), changing P translates these curves horizontally. This 
horizontal axis is the parameter of most direct physical interest. 
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Fig. 3 — Maximum data rate per unit RF bandwidth. 
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Fig. 4 — Relative phase shift per level in one sample interval for optimum systems. 

(ii) n — the optimum number of levels, determined from Fig. 1. 
Here a single universal plot suffices rigorously for all P [That is, without 
small-angle approximations in equation (4)]. 
The vertical axes show: 

Figure 2 — B/N, the bandwidth expansion factor, roughly* one-half 
the ratio of RF to base-bandwidth. This factor varies from about 2 at 
small p or n , to an asymptotic limit of 1 as p, n — > » . For large p, n 
we have small-index phase modulation, with only the first sideband 
significant. Even for the smallest p, n considered here the bandwidth 
expansion is moderate. 

Figure 3 — R /B, the normalized maximum rate in bits per cycle. 
This quantity increases monotonically with p, n . 

Figure 4 — 360 • Aj/N represents the relative phase change in degrees 
corresponding to a change in modulation of one level. 

Figure 5—360 (n — 1) Aj/N represents the maximum relative 
phase change in degrees in one sampling interval, corresponding to a 
change in modulation from the lowest to the highest level. The maximum 
value for this quantity, occurring for the smallest p, n (that is, n = 4) 
is not far from 360°. As p, n increase, the maximum phase change be- 
comes small for optimum systems. 

Within the small-angle approximation, discussed in connection with 
Fig. 1, changing P merely shifts the horizontal (dB) axes of Fig. 1 
and Figs. 2(a) to 5(a). Let us adopt the P = 10" 6 curves as standard, 



* This is because the square-wave modulation assumed here is not strictly band- 
limited; in fact, its spectrum falls off so slowly that its rms bandwidth is infinite. 



FM RATE OPTIMIZATION 



3027 









(a) 








-p 


= 10" 8 






P = to 


-6 ^. 



























I 






(b) 




\ 










\ 





























288°'^ 



216° 



10 20 30 40 50 

10 LOG IO p-RF SIGNAL-TO-NOISE 
RATIO IN DECIBELS 



10 20 30 40 

n -OPTIMUM NUMBER 

OF LEVELS 



Fig. 5 — Maximum relative phase shift in one sample interval for optimum systems. 

and plot the number of dB to be added to the 10 log 10 p axes a sa function 
of P. This is shown in Fig. 6. We remark that this is only an approxi- 
mation, and will begin to fail sooner as P decreases. 

Finally, Fig. 7 compares the maximum data rate for the multilevel 
FM system with the Shannon capacity of the RF channel. The optimum 
data rate ranges from about 19 to 27 percent of the ideal RF channel 
capacity, for error probabilities P between 10 -8 and 10~ 8 . 

We have so far dealt with optimum systems. However, the number 
of levels may be fixed by other constraints, so that suboptimum systems 
are of interest. For example, it may not be practical to have the large 
number of levels required for optimum systems at large RF signal-to- 
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Fig. 6 — Correction for modifying P = 10~ 6 curves to other error probabilities. 
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Fig. 7 — Ratio of maximum data rate to Shannon capacity. 

noise ratios p; we may be restricted to 8 (or 16) levels, and it is necessary 
to determine how much the data rate will be reduced. Now rather than 
maximizing R by varying N and n in equation (1) subject to the con- 
straints of equations (2) and (4), we fix n in equations (5) and (6). 
Figures 8 and 9 show the optimum rate R /B versus 10 log 10 p [given 



2 2.0 
u 
\ 
1/1 

k- 

m 1.6 

i 

crim 

1.2 



0.4 





p = 


!0- 6 


/ 

/ 
/ 


^ 








/ / 
i / 

i / 


^6 






i 














"4 






















n = 2 








NUMBER OF 
LEVELS 

1 



10 20 30 40 50 

10 LOGio P-RF SIGNAL-TO-NOISE 

RATIO IN DECIBELS 



Fig. 8 — Best data rate for suboptimum systems with two, four, and eight levels 
compared to maximum data rate for optimum system. Dashed line^ — maximum data 
rate for optimum system, R /B (see Fig. 3). 
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Fig. 9 — Best data rate for suboptimum systems with two, four, and eight levels 
compared to maximum data rate for optimum system. Dashed lined — maximum 
data rate for optimum system, Ro/B (see Fig. 3). 

also in Fig. 3(a)], together with the rates for two, four, and eight levels, 
determined from equation (6) with n = 2, 4, and 8 for P = 10~ 6 , 10~ 8 in 
equation (4). While eight levels is strictly optimum only at the point 
of tangency between the R 8 and the R n curves, we see that the optimum 
is fairly broad. The corresponding bandwidth expansion factors are 
found from equation (5). 



rv. DISCUSSION 

We have presented the results of Figs. 1 through 9 as continuous 
curves. Actually, only isolated points of these curves are significant, 
since the number of levels must be integral. These continuous curves 
should consequently be replaced by appropriate "staircase" functions, 
but the difference will be significant only for small numbers of levels 
(that is, at low RF signal-to-noise ratios). 

The present theory excludes two- and three-level systems. Naively, 
one might try to extend the present results to these cases by equation 
(17c) and Fig. 5 of Ref. 4. This may not be accurate for the error rates 
considered here (P = 10" 6 , 10" 8 ), because the RF signal-to-noise ratio 
p becomes small, and the basic results of Ref. 4, that is, equations 
(17), (26), and (27), are asymptotic as p becomes large. However, for 
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very much smaller error rates, for example, P « 10~ 30 , it is possible 
that this approach would be productive. 

It would be desirable to extend the present results to binary and 
ternary systems; this will require a different or improved approach 
from the asymptotic evaluation of Ref. 4 for the error probability. It 
seems likely that clicks will dominate the error behavior for optimum 
two- and three-level systems. 

The principal limitation in the present treatment (aside from the 
assumptions of the model, such as a finite-time integrator post-detection 
filter) lies in our lack of knowledge of the precise way in which the basic 
result for the probability of error P (equation (4) above) fails. We have 
merely assumed that this result holds for signal-to-noise ratios down to 
about 10 dB, independently of P or Af/N. This provides additional 
motivation for further study of the asymptotic theory of Ref. 4. 
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